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Abstract 

Prior research in higher education shows that engagement has been inconsistently conceptualized: 
semantic inconsistency has been compounded by variations in the constructs used to operationalize 
engagement. Acknowledging these limitations, we conceptualize student engagement as a multi¬ 
faceted meta-construct, overcoming some of the limitations evident in prior studies. This supports a 
research design that enables us to tap the capacity of the LMS at our institution to operationalize the 
constructs that undergird the Bundrick et al (2014) model. Our inter-cohort analysis reveals significant 
variations in individuals' engagement with the on-line course. Our findings suggest that interactions 
among the primary elements of the learning environment—the student, the teacher, and the content— 
significantly affect engagement and student outcomes. Our findings also suggest an urgent need to 
deepen investigation of student engagement to more fully investigate the dynamics of interaction as 
on-line learning environments account for an increasing proportion of higher education. 
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1. INTRODUCTION 

Quality Matters Standard 5, Learning Interaction 
and Engagement, is based on research on best 
practices for online course design. Each of the 
four guidelines encompassed within this 
standard promote engaging students to become 
active learners, which will contribute to their 
learning process and their persistence (Quality 
Matters, 2011). These guidelines include 
learning activities to promote the achievement 
of the stated learning objectives; learning 
activities to provide for the interaction that 
support active learning, the instructors plan for 
response time and feedback, and explanation of 
requirements for student interaction. 


In the wider context of higher education, 
engagement is seen primarily as a quality of the 
school or college rather than a trait of the 
individual. The amount of time and effort 
students put into their studies and other 
educationally purposeful activities is one of the 
'critical features' of engagement used by Indiana 
University's National Survey of Student 
Engagement (NSSE) to measure collegiate 
quality. NSSE also assesses how the institution 
deploys its resources and organizes the 
curriculum and other learning opportunities to 
enable and encourage students to participate in 
activities that, they argue, research studies 
show are linked to student learning. 
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In this research, we revisit this linkage and 
explore the dynamics of participation in a little 
more depth. Rather than estimates of how 
students spend their time and what they gain 
from attending college - the focus of NSSE - we 
explore engagement at the individual student, 
course and instructor level. This is the locus of 
evolution for engagement. Summary statistics 
mask variations at these levels which, we argue, 
are critical to understanding students' learning 
needs and performance. In this research, we 
have strived to address this limitation by making 
the student the primary unit of analysis. This 
enables us to explore the interactions between 
the three 'core' components of learning - the 
instructor, the material and the student. Both 
our course design and our research design 
emphasize the need for effective scheduling and 
'choreography' of these core components. 

The assignment schedule for the course is one of 
the content elements developed with this need 
in mind: it also follows best practices for 
interaction and thus aligns with Standard 5 of 
the Quality Matters rubric. The rubric outlines 
three types of interaction to promote student 
success within an online course: student-student 
interaction, student-content interaction, and 
student-teacher interaction. With the 

implementation of these best practices for 
student interaction and engagement in place 
within the course, it seems reasonable to expect 
that students should perform well academically if 
they complete the course assessment schedule. 
This paper will test this expectation using 
empirical data to compare student engagement 
and academic achievement in a Systems 
Analysis and Design course over two semesters. 

2. PRIOR RESEARCH 

Prior research in higher education shows that 
engagement has been inconsistently 
conceptualized (Appleton, Christenson, & 
Furlong, 2008; Fredricks & McColsky, 2012). 
Behavioral and psychological indicators (e.g., 
Finn, 1989; Newmann, 1992) have been used in 
various combinations, giving rise to inconsistent 
terminology (Bundrick et al, 2014). Terms such 
as school engagement (e.g., Fredricks, 
Blumenfeld, & Paris, 2004), student engagement 
(Willms, 2003) and academic engagement 
(Libbey, 2004) have been used interchangeably. 
This semantic inconsistency has been 
compounded by variations in the constructs used 
to operationalize engagement. Acknowledging 
these limitations, we conceptualize student 
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engagement as a multi-faceted meta-construct 
that accommodates important distinctions 
among students' evaluations and experiences in 
the various conceptually distinct dimensions of 
their learning environment: behavioral, 

cognitive, and emotional. 

Bundrick et al (2014) articulate these 
distinctions: Behavioral engagement refers to 
various behaviors that are directly oriented to 
learning, such as attending and contributing to 
classes, compliance with rules, completing 
assignments and the time and effort put into 
studying. Cognitive engagement is a little less 
tangible since it involves the student's 
psychological 'investment' in learning and 
mastery of academic materials and their desire 
for challenge. Cognitive engagement is manifest 
in metacognitive strategies such as planning, 
monitoring, and evaluating one's thinking; and 
self-regulation. Emotional engagement refers to 
students' feelings about their relationships with 
others in the learning environment (e.g., 
teachers, peers) and the general sense of 
belonging and connectedness that is often 
derived from such relationships: these inform 
student perceptions of self-efficacy (Compeau et 
al, 1995, 1999) and confidence regarding their 
academic ability. 

Prior research on effective student interaction 
and engagement provides the basis for 
implementation of our course design objectives. 
Student-student interaction, often implemented 
through discussion boards, has been found to 
increase student satisfaction in a course 
(Rothmund, 2008). Additionally, social 

interaction has been found to play a significant 
role in students' sense of learning (Hill et al., 
2009). Davies and Graff (2005) found that the 
level of participation in online discussion boards 
did not correlate with overall course 
performance; however, Rovai and Barnum 

(2003) found that active interaction, the number 
of messages posted by students per week, was a 
significant predictor of perceived learning. 

Nussbaum et al (2004) found that the quality of 
interactions could play an important role in 
student outcomes. Therefore, a set of criteria 
demanding high quality participation, such as 
including sources and providing examples in 
discussion responses, could provide a 

meaningful level of engagement to act as a 
predictor of student success. Sher (2009) found 
that both student-student interaction and 
student-instructor interaction were significant 
contributors to student learning and satisfaction. 
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Prior research on the effectiveness of student 
interaction and engagement in online courses 
has focused primarily on measuring success 
through student perceptions and student 
satisfaction. However, as Bundrick et al (2014) 
have shown, research into engagement among 
students in higher education has been distorted 
through over-abstraction - the assumption that 
summary statistics have a generalizable validity 
to whole populations. 

We address this shortcoming by focusing 
attention on more fundamental measures of 
personal interaction within the core learning 
components. We aim to develop a more causally 
robust linkage between engagement and 
performance and, in so doing, propose indicators 
of engagement that are more immediately 
relevant to students, instructors and course 
designers in the rapidly evolving milieu of on¬ 
line core learning components. 

3. RESEARCH SETTING 

The University of South Florida St. Petersburg 
(USFSP) offers a range of distinctive graduate 
and undergraduate programs in the arts and 
sciences, business, and education within a close- 
knit, student-centered learning community that 
welcomes individuals from the region, state, 
nation and world. We conduct wide-ranging, 
collaborative research to meet society's needs 
and engage in service projects and partnerships 
to enhance the university and community's 
social, economic and intellectual life. As an 
integral and complementary part of a multi- 
institutional system, USF St. Petersburg retains 
a separate identity and mission while 
contributing to and benefiting from the 
associations, cooperation, and shared resources 
of a premier national research university. The 
university's online learning is delivered through 
a learning management system; Canvas by 
In structure. 

The recent adoption of Quality Matters (Quality 
Matters, 2013), an online course quality 
management program, at USFSP has provided a 
set of specific standards that can be used to 
enhance the accessibility of courses. Quality 
Matters is a quality assurance program that 
facilitates a peer review process to recognize 
courses that follow best practices for design and 
promote student success in online education. 
Courses are reviewed using a rubric (Quality 
Matters, 2011) comprising a set of eight 
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research-based standards for design that heavily 
promote student engagement. 

4. RESEARCH DESIGN 

The review above prompts us to posit that 
students who display higher levels of 
engagement perform academically at a higher 
level than those who display lower levels of 
engagement. In this initial research, we explored 
this relationship using a robust but rather 
rudimentary analysis which enabled us to 
directly compare two cohorts of students in the 
Systems Analysis and Design course. 

The complexity and conceptual richness of the 
design artifacts and process taught and assessed 
in the Systems Analysis and Design course 
provides a broad and diverse range of 
opportunities for engagement (Avison and 
Fitzgerald, 2006; Topi et al, 2010). The learning 
outcomes are both a prerequisite for and a 
predictor of success in the major (Kmetz & 
Davis, 2014): they have also been aligned with 
the skills indicative of mastery of the language, 
tools and techniques that enable their effective 
use in employment (Yourdon, 1993; Topi et al, 
2010 ). 

Professional as well as instructional experience 
in this highly applied field highlights engagement 
as a 'critical success factor' for the synthesis of 
the technical skills and cognate knowledge. 
Although by no means unique, the 
interdependence of these learning outcomes and 
the core learning components on which they 
depend place students who disengage at 
substantially higher risk of failure than those 
pursuing more didactic courses. 

Bundrick et al (2014) consider how the primary 
elements of the learning environment—the 
student, the teacher, and the content—interact 
to affect engagement, and propose a conceptual 
model that highlights the interdependence of 
these core elements and interactions. 

Our research design operationalizes this model: 
however, our units of analysis are rather coarse, 
limited by the range of metrics provided by the 
LMS. Following Bundrick et al (2014) we 
identified a number of surrogate metrics to 
operationalize the constructs underlying 
engagement: data for the analysis was drawn 
directly but anonymously from the course 
analytics provided by the LMS. These include 
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page views, participations, and assignment 
submissions. 


Student 


Relationships 





Teacher 


Relevance 


Competence 


Content 


Figure 1 The Student Engagement Core Model 
(Bundrick et al, 2014) 

Page views are calculated for each student from 
the day that the course opens to the day that 
the course closes. Canvas records a page view 
anytime a student lands on a page within the 
course. Course pages include Home, Syllabus, 
module instruction pages, and assignment 
instruction pages. 

In addition to page views, Canvas analytics 
calculates cumulative participation counts for 
each student. A participation is recorded any 
time a student submits a quiz, starts taking a 
quiz, submits an assignment, creates a wiki 
page, posts a new comment to a discussion, 
joins a web conference, or loads a collaboration 
to view or edit the document. 

The analytics also provides an overview of the 
student's assignment submissions over the 
semester. It provides a count of the on-time 
submissions, late submissions, and missing 
submissions. 

The first section of this course was offered 
during a six-week session during Summer 2013. 
The second section of was offered during a full- 
length semester in Fall of 2013. Both sessions 
were offered completely online and shared a 
common teaching and assessment model 
comprising 24 assignments. 



Figure 2 Example view of student analytics. 

The assessment regime is comprehensive, 
designed using the pedagogic and evaluative 
guidance provided by Topi et al (2010). The 
synthesis of cognate, analytic and design 
competence responds both to this curricular 
guidance and the trends through industry and 
employment analyses (Avison et al, 2006). 



Figure 3 Assessment 'regime' (see also 
Appendix A) 

Opportunities for engagement in this course are 
designed to promote student-student, student- 
content, and student-instructor interaction. The 
assessment schedule includes a discussion board 
to allow students to introduce themselves and 
meet their classmates. This creates a community 
of learning within the course. Each of the 13 
learning modules includes a low-stakes quiz that 
acts as a knowledge check for the students. To 
assess students' comprehension and application 
of the material there are two discussion boards 
centered around important topics in the field and 
four portfolio assignments allowing students to 
demonstrate their ability to complete major 
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tasks within a systems analysis and design 
project. The summative assessment for this 
course includes two exams; a midterm and final. 
Both exams consist of two parts; a multiple 
choice portion and an essay portion. Each 
module also includes lectures, examples, and 
readings. 


other 'outcome' is not clear. However, it seems 
reasonable to infer that those in this cohort with 
failing grades chose not to continue. The brevity 
of the semester gave students an opportunity to 
drop the class without academic penalty just a 
few days before its end: this skews the sample 
significantly. 


5. FINDINGS 

The Summer 2013 Section was completed by 34 
students: the table below summarizes the 
surrogate metrics that we used to assess their 
levels of engagement and the resultant leaning 
outcomes. The table sets the data out in broad 
grade-based classes (rows) to maintain cohesion 
in our commentary, providing consistency with 
the performance and outcome metrics used in 
the research reported in Section 2. This also 
provides a familiar basis for comparison with 
grading schema used to report learning 
outcomes at other institutions. 


Final Grade Range 

Average Page Views Per 

Student 

Average Participations Per 
Student 

Average Assignments 
submitted on time per student 

Average assignments 
submitted late per student 

Average assignments missing 
per student 

90% - 100% 

(6 students) 

548.00 

35.30 

23.80 

0.20 

0.00 

80% - 89% 

(25 students) 

601.80 

32.90 

23.10 

0.10 

0.80 

70% - 79% 

(3 students) 

713.70 

26.30 

20.30 

1.70 

2.00 

Below 70% 

(0 students) 

Not 

applicable 

Not 

applicable 

Not 

applicable 

Not 

applicable 

Not 

applicable 


Table 1 Summer 2013 Engagement and 

Outcomes 


Such discontinuance clearly affects the grade 
distribution. Recent work by Munro (2014) 
highlights other significant characteristics of 
such disengagement: often condoned by 
instructors, it gives rise to grade inflation and, 
he argues, is a form of fraud. 

Nevertheless, Figure 4 provides insight into the 
relationship between outcomes and engagement 
as measured through participation in the 
assessment 'regime' set out in Figure 3. To 
provide a benchmark to interpret these data, it 
should be noted that there are 24 assessments 
that students participate in: of those, 18 provide 
one opportunity to participate (the quizzes and 
modeling submissions that build the student 
portfolio). The discussions, on the other hand, 
are more interactive. The two graded 
discussions require multiple participations: the 
first to 'post' and 'reply'; the second to 'post' 
and then reply to each of two 'threads'. In 
combination, the 24 participations span a range 
of interactions and thus provide the multi¬ 
faceted indication of engagement that Bundrick 
et al (2014) propose. 


Average Participations 


40 



Table 1 shows the relationship between 
participation and outcome: note that no 
students were assessed at below 70% in this 
cohort. The cohort size in this short six-week 
'semester' was reduced significantly (from an 
initial 40 students) immediately prior to the 
'drop' date. The dynamics of the relationship 
between students' anticipation of success or 


Figure 4 Summer 2013 Participation 

A differently nuanced indicator of engagement is 
provided through measurement of the number of 
page views. Although less substantive than 
measures of participation - since they do not 
discriminate between mere browsing and more 
intellectually engaged reading and assimilation - 
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their overall volume provides a complementary 
scale and arguably more normalized measure of 
engagement. 

Overall, students in this cohort engaged with the 
material 621 times or about 15 times per day 
during the 6 week duration of the course. Figure 
5 shows that the standard deviation in this much 
larger data sample is quite low: the difference 
between the highest and lowest columns is only 
166. 


Average Page Views: Summer 2013 


800 



received is inverse. In the context of 
engagement, this seems significant, suggesting 
that participation is a more reliable indicator of 
outcome than page views. 

We turn now to consider two more assessment 
indicators of engagement. Figure 7 shows the 
average counts for timely submission of the 
twenty four assignments (see Figure 3). Here, a 
more direct relationship is immediately clear: 
timely submission of assignments correlates 
positively with outcomes. 


Assignments submitted on time 



Figure 5 Summer 2013 Page views 



Figure 6 Summer 2013 Outcomes 

Two observations can be made here: firstly, 
classification of the data using outcomes masks 
internal variations. Our observations while the 
course was live showed significant variations 
within each of the broad categories in Figure 4. 
Our current research strives to make these 
episodic variations more tractable. Secondly, the 
trend of the data seems counterintuitive: the 
relationship between number of page views and 
quality of outcome as indicated by the grade 


Figure 7 Summer 2013 Assignment 
timeliness 

As might be anticipated, late submission of 
assignments correlates negatively with 
outcomes. Although significantly smaller, this 
data set shows remarkable symmetric 
consistency with those in Figure 8. 


Assignments missing 

1 c 

9 
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Figure 8 Summer 2013 Assignment 
completeness 
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Considering Figures 5, 6, 7 and 8 together, the 
data present apparently contradictory 
relationships between the level of engagement 
with the materials overall and outcomes 
(compare Figures 5 and 6) and that between 
engagement with the assignments (compare 
Figures 6, 7 and 8). 

The metrics for Fall 2013 are summarized in 
Table 2: they show the engagement and 
outcomes for the (larger) 41 student cohort and 
allow us to explore this apparent contradiction a 
little further. 


Final Grade Range 

Average Page Views Per 
Student 

Average Participations Per 
Student 

Average Assignments 
submitted on time per 
student 

Average assignments 
submitted late per student 

Average assignments 
missing per student 

90% - 100% 

(4 students) 

811.50 

49.00 

24.00 

0.00 

0.00 

80% - 89% 

(24 students) 

611.30 

42.40 

23.40 

0.60 

0.00 

70% - 79% 

(8 students) 

483.00 

36.10 

23.10 

0.30 

0.60 

Below 70% 

(5 students) 

283.40 

13.40 

7.20 

1.00 

15.80 


Table 2 Fall 2013 Engagement and 

Outcomes 

The cohort size in this full semester was more 
stable than Summer 2013, although large for 
such a technical course. 


Average Participations 


60 



Figure 9 Fall 2013 Participation 


Figure 9 shows a strong correlation between 
outcomes and engagement as measured through 
participation in the assessment 'regime' set out 
in Figure 3. Participation rates are high: an 
average of 49 participations for students in the 
top performance category. 

Students in the lowest category show 
substantially lower engagement, participating on 
average only 27% of the level of those achieving 
the highest grade outcome. 


Average Page Views 


1000 



Figure 10 Fall 2013 Page views 

Overall, students in this cohort engaged with the 
material 547 times. This represents a lower 
'density' of engagement in this longer 15 week 
semester at only 5 views per day. This is 
consistent with the number of student 'effort 
hours' expected at this institution: 5 views per 
day when 10 hours of effort per week are 
expected equates to about 12.5 views per day in 
a six-week semester - such as the one analyzed 
above - where 25 hours effort per week are 
anticipated. 

Figure 10 also reveals a higher standard 
deviation for the longer course: the difference 
between the highest and lowest columns is 
significant at 527. On average, higher 
performing students viewed the course content 
three times more frequently than those in the 
lowest category. 

Again it should be remembered that our 
classification of data using outcomes masks 
internal variations. Our observations while the 
course was live showed similar significant 
variations within each of the broad categories in 
Figure 10. The highest and lowest number of 
views for the highest outcome class was 1102 
and 585: the low is less than half of the high. 
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For those who achieved a B (80-89%) the 
figures were 996 and 339: the low is a little over 
one third of the high. This is consistent with the 
size of this group (24) in our outcome 
distribution. Those who achieved a C (70-79%) 
ranged from 754 to 476 page views: a much 
narrower range - again a function of the group 
size (8 students). 

The lowest performing group included two 
students who did not complete the course. The 
range is large here: from 669 to 40 page views. 
Nevertheless, the overall trend of the data for 
the Fall 2013 cohort seems more intuitive: the 
correlation between number of page views and 
outcome is positive. 



Turning to our assignment-oriented indicators of 
engagement, we see that Figure 12 reveals a 
similarly direct correlation between submission 
timeliness for the twenty four assignments (see 
Figure 3) and outcomes. 


Assignments submitted on time 


30 



Figure 12 Fall 2013 Assignment 
Submission 


Again we see some intuitive complementarity in 
the distributions of assignments submitted on 
time and those missing completely. However, 
comparison of Figures 12 and 13 shows a 
stronger 'fall off' between those achieving 
passing grades and those who didn't. 

The apparent contradiction seen when Figures 5, 
6, 7 and 8 (for the Summer cohort) are 
considered together is not evident when Figures 
10-13 are compared. For the Fall 2013 cohort 
the relationships between the level of 
engagement with the materials overall and 
outcomes (Compare Figures 9 and 11) are more 
consistent with that for engagement with the 
assignments (Compare Figures 11, 12 and 13). 


Assignments missing 

20 



Figure 13 Missing Assignments Fall 2013 


6. COMMENTARY 

Our analysis uses metrics that measure some of 
the interactions in the Bundrick et al (2014) 
Student Engagement Core Model (Figure 1). 
Despite the limitations that we discuss below, 
the data provide coherent and consistent 
comparisons of student engagement. The overall 
similarity between the data sets for the two 
cohorts highlights this. It also highlights the 
significant difference in the patterns of 
engagement between the two cohorts as 
measured by page views. 

Figure 14 shows opposing trends in the relation 
between page views and outcome. The blue bars 
show engagement for the Summer cohort. The 
correlation between page views and outcomes is 
negative. These summary data again disguise 
intra-class variation. 
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1000 



90-100% 80-89% 70-79%Below 70% 

■ Average Page Views Per Student Summer 
* Average Page Views Per Student Fall 


Figure 14 Engagement and Outcomes: two 
semester comparison 

The highest and lowest number of views for the 
highest outcome class was 1003 and 366. Here, 
the low is close to one third of the high, a 
significantly larger proportion than that seen 
among the equivalent group in the fall cohort. 
For those who achieved a B (80-89%) the 
figures were 951 and 292: here the low is less 
than one third of the high (31%). This is 
consistent with both the size of this group (25 
students) in the outcome distribution and - 
perhaps significantly - the range in the higher 
performing group. Those in the Summer cohort 
who achieved a C (70-79%) ranged from 952 to 
590 page views: two observations are pertinent 
here. Firstly, the range among the students 
(n = 3) in this group is narrow, the low being 
almost two thirds (62%) of the high. Secondly - 
and perhaps consequently - the average for this 
group is closer to that for the other two outcome 
groups in this cohort. This gives a much lower 
gradient to the trend in Figures 5 and 14. 

The trend for the Fall 2013 cohort (Figure 10 
and the red bars in Figure 14) has an inverse 
gradient but one which - more intuitively - 
indicates that performance decreases as page 
views decrease. 

Page views among students in the 80-89% 
outcome class are remarkably similar, both in 
number and range. For the Summer cohort, the 
low of 292 is 31% of the high (951). For the Fall 
cohort, the low is 34% (339) of the high (996). 

Although the data for this inter-cohort 
comparison do not immediately reveal a causal 


link between page views and outcomes, the 
inconsistencies are significant and raise a 
number of questions. Although, on the face of it, 
it might be argued that participations and 
assignment submission figures provide more 
reliable prediction of outcomes, this deflects 
from the potential that these finer-grained data 
contribute to our understanding of the dynamics 
of student engagement. 

7. CONCLUSIONS 

In this work we have strived to overcome some 
of the limitations evident in prior studies of 
student engagement. By conceptualizing 
engagement as a multi-faceted meta-construct 
we developed a research design that enabled us 
to explore the capacity of the LMS at our 
institution to operationalize the constructs that 
undergird the Bundrick et al (2014) model. 

Clearly, our study is limited by a number of 
factors: the single course comparison and the 
'coarseness' of the data provided by the LMS, 
both in terms of the range of scales measures 
and the limited number of time-points at which 
they are reported. 

Nevertheless, this initial research highlights the 
potential of the cohort and individual (student) 
units of analysis to provide insights into 
engagement that substantially supplement the 
collective institution-level analyses presented by 
NSSE. 

Although 'complete', this research presents more 
questions than answers: we are currently 
working with our LMS providers and 
administrators to extract data that will provide 
an even more dynamic view of the interactions 
at the heart of the Bundrick et al (2014) model 
(Figure 1 above). This will enable our future 
work to address the limitations set out above 
and the questions posed. In particular, we will 
use the more comprehensive data to relate 
student interactions, engagement and outcomes 
in the on-line 'classroom' setting to examine the 
level and significance of inter-student 
engagements noted by Baepler & Walker (2014). 
The research question here is whether 
participation in collaborative assignments such 
as those designed for the course discussed in 
this paper allow more full 'exploitation' of the 
opportunities that 'new classrooms' present and 
thus positively affect student outcomes. 
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Figure 3 Assessment Regime 
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